Advancement of information technology has a wide implementation of networked systems where businesses of various sizes utilize multiple computing systems to facilitate their operations. The use can range from simple event logging to database management and analytics. As more operations are computerized and businesses grow in size, the volume of available data is rapidly increasing and frequently overwhelming. Moreover, different networked systems may use different data types and formats, making it difficult for business owners and managers to understand the vast amount of data and make appropriate decisions. Effective and efficient management of such vast amount of data can provide significant competitive advantage.
Another complicating factor arises in collection and storage of such data, where networked systems are susceptible to unexpected problems such as network-wide outages or system level failures. These circumstances are detrimental to the collection and storage of data from the networked systems because network outages can prevent all data from being transferred from one system to another and system level failures can result in data loss until the problems are resolved. Prior art systems have not been able to account for such failures, skipping data aggregation if a networked system is unavailable or being unable to resume aggregation from the last successful aggregation.
Still further, the data collection is also without merit if the data collection and analysis occur over a long period of time. “A long period” is a relative term where even a 10-minute delay in the collection and analysis may be too long in some circumstances while other systems may be okay with collecting data only once per day. As business operations advance to require more rapid responses, however, a real-time or a near real-time data collection and analysis become more important.
Therefore, there is a need for dynamic aggregation of data in near real-time from different networked systems that can collect data of different formats and types, reconciling them to a single format to support sophisticated analysis, while being robust enough to account for unexpected problems and resume collection once the problems are resolved.